1. Statement of the Technical Field
The inventive arrangements concern image registration, and more particularly registration of two-dimensional image data acquired by electro-optical image sensors with three-dimensional point cloud image data acquired by LIDAR sensors.
2. Description of the Related Art
Imaging sensors are used in a wide variety of applications, including homeland security and military applications. For example, sensors of this type are commonly positioned on aircraft and earth-orbiting satellite platforms. Conventional electro-optical (EO) sensors have long been used for collection of such image data and generally produce two dimensional data. Such data generally corresponds to a projection of the image onto a planar field which can be entirely defined by an x and y coordinate axis.
More recently, there has been a growing interest in three-dimensional imaging data Those skilled in the art will appreciate that there are a variety of different types of sensors, measuring devices and imaging systems in existence which can be used to generate 3D point cloud data. One example of a 3D imaging system that generates one or more frames of 3D point cloud data is a conventional LIDAR imaging system. In general, such LIDAR systems use a high-energy laser, optical detector, and timing circuitry to determine the distance to a target. In a conventional LIDAR system one or more laser pulses is used to illuminate a scene. Each pulse triggers a timing circuit that operates in conjunction with the detector array. In general, the system measures the time for each pixel of a pulse of light to transit a round-trip path from the laser to the target and back to the detector array. The reflected light from a target is detected in the detector array and its round-trip travel time is measured to determine the distance to a point on the target. The calculated range or distance information is obtained for a multitude of points comprising the target, thereby creating a 3D point cloud. The 3D point cloud can be used to render the 3-D shape of an object.
Each point in the 3D point cloud is somewhat analogous to the pixel data generated by a digital camera, except that the 3D point data is arranged in three dimensions, with points defined at various locations by an x, y, and z coordinate axis system. The 3D image or cloud of points produced by the LIDAR sensor is commonly known as point cloud data. In contrast to 2D pixel data, the x,y,z coordinates provide only spatial information whereas the 2D pixels contain visual intensity information as a function of wavelength. It should be noted that some LIDAR sensors can simultaneously collect intensity data by long dwell times over a region of interest.
Point cloud data is useful for creating 3D models of a scene. However, it often lacks much of the detailed visual information normally associated with image data acquired using conventional EO sensors. Accordingly, it is advantageous to combine 2D EO imaging data with 3D point cloud data for the same scene. However, combining the two different sets of imaging data necessarily requires an image registration step. Such image registration step is usually aided by metadata associated with image. For example, such metadata can include 1) orientation and attitude information of the sensor and 2) latitude and longitude coordinates associated with the corner points of the image, and 3) in the case of point cloud data, the raw x, y, and z point locations for the point cloud data. This metadata can be used to determine the overlap region between the data sets as well as the correspondence point search reach area. Still, this image registration step can be difficult and time consuming because it requires precise alignment of the EO and LIDAR image data acquired by different sensors at different data collection times and different relative sensor positions. Moreover, the point cloud data is fundamentally different as compared to the EO image data, making for a more complex cross-sensor registration problem.
Various registration schemes have been proposed to solve the foregoing registration problem. For example, some registration schemes make use of a LIDAR intensity image as an aid in the registration process. The LIDAR intensity image is obtained by causing the LIDAR sensor to dwell on a scene for an extended period of time. Reflected intensity data is collected by the sensor during this period of time. This results in a 2D image data set obtained by the LIDAR sensor that is somewhat more analogous to the conventional image data collected by the EO sensor. The LIDAR intensity image can thereafter be used in various methods to facilitate the registration process Still, registration using a 2D LIDAR intensity image has many drawbacks. First, it only provides a 2D registration of the LIDAR data to the 2D EO image. Second, using this 2D LIDAR intensity image along with the raw 3D point cloud data and the 2D EO image can be processing intensive and involve a substantial amount of computer memory.
Other 2D to 3D registration techniques are also limited in the sense that they are primarily useful only with urban scenes (and require shape information) or scenes involving medical imagery (and require markers on the 2D image). Such systems also require a good initial guess of the alignment parameters associated with the registration.